Adjust defaults for multimodal-related parameters to work for default deployments #2717

pamelafox · 2025-09-08T20:15:42Z

Purpose

Our frontend was setting the multimodal-parameters sendImageSources and searchImageEmbedding parameters to true, which caused an error in backends with multimodal disabled.

This corrects the frontend to default to False AND adds an extra guard to backend to ensure they can only be set to true if multimodal is enabled.

Also made sure /ask supports send_text_sources like /chat does, the logic was missing.

Test wise, I adjusted an e2e test to confirm the defaults, and added two backend tests.

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[ ] Yes
[X] No

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[X] No

Type of change

[X] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

The current tests all pass (python -m pytest).
I added tests that prove my fix is effective or that my feature works
I ran python -m pytest --cov to verify 100% coverage of added lines
I ran python -m mypy to check for type errors
I either used the pre-commit hooks or ran ruff and black manually on my code.

pamelafox · 2025-09-08T20:20:23Z

app/backend/approaches/approach.py

-                text_sources.append(f"{citation}: {nonewlines(' . '.join([cast(str, c.text) for c in doc.captions]))}")
-            else:
-                text_sources.append(f"{citation}: {nonewlines(doc.content or '')}")
+            if include_text_sources:


The only change here is the very top line-

pamelafox · 2025-09-08T20:21:15Z

app/backend/approaches/approach.py

-        if download_image_sources:
-            return DataPoints(text=text_sources, images=image_sources, citations=citations)
-        return DataPoints(text=text_sources, citations=citations)
+        return DataPoints(text=text_sources, images=image_sources, citations=citations)


This makes image_sources [] when not including image sources, which is consistent with text_sources being [] when not including text sources. Snapshots already aligned with this, and it simplified code.

pamelafox · 2025-09-08T20:22:51Z

app/frontend/src/pages/ask/Ask.tsx

-                // Set default vector field settings
-                setSearchTextEmbeddings(true);
-                setSearchImageEmbeddings(true);
+                // Initialize from server config so defaults follow deployment settings


This is how /chat was already doing it, fixed for consistency

Copilot

Pull Request Overview

This PR fixes multimodal-related parameter defaults to prevent errors in deployments without multimodal features enabled. The frontend was incorrectly defaulting multimodal parameters to true, causing backend errors when multimodal was disabled.

Key changes:

Updated frontend defaults for sendImageSources and searchImageEmbeddings from true to false
Added backend guards to ensure multimodal parameters can only be enabled when multimodal is actually available
Fixed missing send_text_sources support in the /ask endpoint to match /chat functionality

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
app/frontend/src/pages/chat/Chat.tsx	Changed default values for image-related parameters from true to false
app/frontend/src/pages/ask/Ask.tsx	Changed default values for image-related parameters from true to false and updated initialization logic
app/backend/approaches/chatreadretrieveread.py	Added multimodal guards and support for send_text_sources parameter
app/backend/approaches/retrievethenread.py	Added multimodal guards and support for send_text_sources parameter
app/backend/approaches/approach.py	Updated get_sources_content method to handle include_text_sources parameter
tests/test_app.py	Added tests for send_text_sources and multimodal parameter validation
tests/e2e.py	Updated e2e test to validate correct default parameter values

Copilot · 2025-09-08T20:25:10Z

app/backend/approaches/approach.py

-        if download_image_sources:
-            return DataPoints(text=text_sources, images=image_sources, citations=citations)
-        return DataPoints(text=text_sources, citations=citations)
+        return DataPoints(text=text_sources, images=image_sources, citations=citations)


The return statement always includes image_sources even when download_image_sources is False. This should conditionally include images only when download_image_sources is True, otherwise pass None or an empty list for the images parameter.

Suggested change

return DataPoints(text=text_sources, images=image_sources, citations=citations)

return DataPoints(

text=text_sources,

images=image_sources if download_image_sources else None,

citations=citations

)

@copilot But it will be an empty list in that case, since image_sources is only populated when download_image_sources is True?

mattgotteiner · 2025-09-08T21:00:51Z

app/backend/approaches/chatreadretrieveread.py

        search_index_filter = self.build_filter(overrides, auth_claims)
        send_text_sources = overrides.get("send_text_sources", True)
-        send_image_sources = overrides.get("send_image_sources", True)
+        send_image_sources = overrides.get("send_image_sources", self.multimodal_enabled) and self.multimodal_enabled


pamelafox added 2 commits September 8, 2025 12:22

Fix frontend defaults and e2e test

1f00d35

Add backend guards and tests

61c500d

pamelafox requested review from Copilot and mattgotteiner September 8, 2025 20:18

This comment was marked as outdated.

Sign in to view

pamelafox commented Sep 8, 2025

View reviewed changes

Deault send_image_sources to self.multimodal_enabled

e64e72a

pamelafox commented Sep 8, 2025

View reviewed changes

pamelafox requested a review from Copilot September 8, 2025 20:24

Copilot AI reviewed Sep 8, 2025

View reviewed changes

mattgotteiner reviewed Sep 8, 2025

View reviewed changes

mattgotteiner approved these changes Sep 8, 2025

View reviewed changes

pamelafox merged commit 74b66bd into Azure-Samples:main Sep 8, 2025
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adjust defaults for multimodal-related parameters to work for default deployments #2717

Adjust defaults for multimodal-related parameters to work for default deployments #2717

Uh oh!

pamelafox commented Sep 8, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

pamelafox Sep 8, 2025

Uh oh!

pamelafox Sep 8, 2025

Uh oh!

pamelafox Sep 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Sep 8, 2025

Uh oh!

pamelafox Sep 8, 2025

Uh oh!

mattgotteiner Sep 8, 2025

Uh oh!

Uh oh!

Uh oh!

Adjust defaults for multimodal-related parameters to work for default deployments #2717

Adjust defaults for multimodal-related parameters to work for default deployments #2717

Uh oh!

Conversation

pamelafox commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Does this introduce a breaking change?

Does this require changes to learn.microsoft.com docs?

Type of change

Code quality checklist

Uh oh!

This comment was marked as outdated.

Uh oh!

pamelafox Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

pamelafox Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

pamelafox Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

pamelafox Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

mattgotteiner Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pamelafox commented Sep 8, 2025 •

edited

Loading